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Method and apparatus for signal processing, computer program product, computing system 
and camera 



The invention regards a method for signal processing, wherein a sensor signal 
of an image sensor is provided as an input and wherein the input is reconstructed in a filter to 
establish an output for further processing, wherein the filter comprises at least one 
reconstraction-filter selected from the group consisting of: a luininance-reconstruction-filter, 
5 a red-green-blue-color-reconstruction-filter and a contour-reconstruction-filter, wherein the 
input comprises a plurality of pixels and a pixel provides a color value assigned to at least 
one of the colors red, green or blue. Hie invention also regards an apparatus for signal 
processing which is in particular adapted to execute the method; comprising an image sensor 
for providing a sensor signal as an input and a filter for reconstructing the input to establish 

10 an output for further processing, wherein the filter comprises at least one reconstruction-filter 
selected from the group consisting of: a luminance-reconstruction-filter, a red-green-blue- 
color-reconstruction-filter and a contour-reconstruction-filter, wherein the input comprises a 
plurality of pixels and a pixel provides a color value assigned to at least one of the colors red, 
green or blue. Further the invention regards a computer program product, a computing 

1 5 system and a camera adapted for signal processing. 

Digital cameras based on digital signal image sensoring of e.g. video and still 
images may be advantageously equipped with an image sensor comprising a red-green-blue 
(RGB) Bayer-color-filter-array. In such RGB Bayer-color-filter array each pixel senses a red, 
green or blue primary color in a predefined pattern. This pattern is built up of alternating 

20 green/red columns and green/blue columns. Such a sensor may have a limited resolution in 
comparison with a camera using a separate image sensor for each primary color. However, a 
camera with three image sensors has three times as much pixels contributing to the resolution 
than a single RGB Bayer-sensor. Using three sensors is for most applications 
disadvantageous due to cost and size requirements of the application. On the other hand, 

25 when using one single image sensor to sense all three primary colors red, green and blue 

within one single array, advantageously in a RGB Bayer-color-filter array, it is necessary to 
reconstruct missing pixels of certain colors to process a consistent whole of a picture. Due to 
the RGB Bayer-structure, different Nyquist-domains respectively with regard to green, red 
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and blue colors result in a color dependent resolution and possibly aliasing patterns. 
Nevertheless the RGB Bayer-structure is one of the best performing signal color arrays. 



Several interpolation schemes may be provided to increase signal quality. In 



5 WO99/39504 a conventional method of interpolation is describ3ed in rather general terms 
wherein an intermediate color signal is interpolated at positions where no signal of a given 
color is present and an average of a given color is generated. 



scheme as described in the WO99/04555 and in the European patent application with the 
10 application number EP 01 200 422.2 not yet published. However such methods still suffer 
from e. g. aliasing of a free luminance signal or further signal distortion. Such signal 
distortion especially results in an erroneous generation of false colors in an image. 



sensors has been described merely concerns the green color reconstruction. The red and blue 

15 colors remain reconstructed in a conventional way. Merely a missing green pixel is 

reconstructed. The reconstruction of the missing green pixel is carried out by means of a 
median filter sorting three specific variables: two of them are derived from the green color, 
the third one from the red or blue color. A disadvantage of this method is, that for high 
saturated colored edges artifacts are introduced which look like the border of a postage 

20 stamp. The algorithm disclosed in WO 99/04555 will be referred to as the smartgreenl- 

* algorithm. The smartgreenl -algorithm is based on the concept that resolution losses are best 
observed at high frequencies near white scene parts and less good near colored parts. 
Keeping this in mind, the contribution of the red and blue pixels is used to help determining 
the reconstruction value of the missing green pixels. The object of the smartgreenl- 

25 reconstruction is to maximize the resolution of the green color. For this purpose the median 
filter algorithm is applied as follows: naturally, the location occupied by a red (R) or blue (B) 
pixel is a location of a missing green pixel. In the smartgreenl -reconstruction-algorithm a 
center value of a 3 x 3 pixel array, also called the median value, is applied for the 
reconstruction of the missing green pixels. Consequently, the simple median filter for green 

30 merely replaces a conventional interpolation concept of green reconstruction, whereas the 
conventional red and blue reconstruction method is maintained to be a simple interpolation. 
Luminance filtering, color filtering and contour filtering is also restricted to the filtering for 
green. False color detection is merely based on a conventional interpolation concept of green 



A further method for signal processing uses a more advantageous interpolation 



In WO 99/04555 a green reconstruction method a for RGB Bayer image 
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reconstruction, and also the conventional red and blue reconstruction is maintained to be a 
simple interpolation. 

Such smartgreenl -reconstruction method improves the resolution of the green 
pixels in the horizontal and vertical direction with the aid of the information of a red and/or 
blue pixel. This conventional method relies on interpolating the color sample to be 
interpolated in dependence upon neighboring color samples of the same color and a 
differently colored sample merely from the same location. As a consequence, the 
reconstructed signal suffers from red-and/or blue-colored aliasing. Vertical and horizontal 
colored edges suffer from a green intensity modulation in the respective direction resembling 
the border of a postage stamp. 

Further improvement as outlined in EP 01200422.2, herein referred to as the 
smartgreen2-reconstruction-algorithm, was able to significantly improve resolution but not to 
remove the mentioned disadvantages of signal distortion and signal aliasing. In particular at 
edges and high frequencies some signal distortion is still visible such as alternating colors 
with neighboring pixels. Artifact black and also white dots are generated erroneously. 

Parallel processing of a color- and a contour-reconstruction filter in principle 
is known from WO99/04554. However, this conventional concept suffers from the same 
disadvantageous as outlined above. 



This is where the invention comes in, the object of which is to specify a 
method and apparatus for signal processing, and also a computer program product for signal 
processing, a computing system and a camera adapted for signal processing such that signal 
quality is improved. In particular, a signal should be improved with regard to signal 
distortion, and aliasing, but the signal should still provide sufficient resolution. 

As regards, the method the object is achieved by a method as mentioned in the 
introduction wherein in accordance with the invention the method further comprises the steps 
of: 

- applying the luminance-reconstruction-filter to an array of pixels of predetermined array 
size comprising a number of pixels, wherein at least one of the number of pixels is 
formed by a red-pixel assigned to the color of red, at least one of the number of pixels is 
formed by a blue-pixel assigned to the color of blue, and at least one of the number of 
pixels is formed by a green-pixel assigned to the color of green and 

- weightening the red- and/or the blue-pixel by a green-parameter, and 

- summarizing the pixels of the array into one output-pixel, and 
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- centering the output-pixel in the array, and 

- applying the contour-reconstruction-filter in parallel with the color-reconstruction-filter. 

As regards the apparatus the object is achieved by an apparatus as mentioned 
5 in the introduction wherein in accordance with the invention it is proposed that 

- the reconstruction-filter is adapted to be applied to an array of pixels of predetermined 
array size comprising a number of pixels, wherein at least one of the number of pixels is 
formed by a red-pixel assigned to the color of red, at least one of the number of pixels is 
formed by a blue-pixel assigned to the color of blue, at least one of the number of pixels 

10 is formed by a green-pixel assigned to the color of green and the apparatus is further 

comprising: 

- means for weightening the red- and/or the blue-pixel with a green-parameter. 

- means for summarizing the pixels of the array into one output-pixel, and/or means for 
centering the output-pixel in the array, and 

15 - means for parallel processing of contour- and color-reconstruction. 

The term pixel herein is used in particular to refer to the value of a color 

sample in the signal. 

The present invention has arisen from the idea to provide a concept of flexible 
design for the reconstruction of images. The main idea is to provide concept of reconstruction 

20 based on a white compensated luminance-reconstruction. This idea basically is realized by 
parallel processing of color-reconstruction and contour-reconstruction. Weightening the red- 
and/or blue-pixel with a green-parameter is a further major improvement, whereas prior art 
concepts merely rely on the reconstruction of a missing green-pixel. The proposed 
reconstruction-filters are designed to be applied to an array of pixels of predetermined array 

25 size. Consequently filtering is performed in an advantageous way on this array. Whereas 

conventional methods rely on simple interpolation in dependence upon neighboring samples 
or a sample of the same location, the proposed concept provides specifically adapted 
reconstruction-filters which take into account all pixels of the array. 

Parallel contour processing for Bayer image sensors allows to use the green 

30 signal from the image sensor for generating a two-dimensional contour-signal in parallel with 
the RGB-color-reconstruction. Advantageously no extra row delays are needed as it is the 
case with serial contour processing. Further advantages will be outlined in chapter 2 of the 
detailed description. 
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In a further configuration advantageously a contour-reconstruction-filter is 
applied, in particular parallel to the luminance-reconstruction-filter and their reconstruction 
signals are added. The contour-reconstruction-filter advantageously is applied to an array- 
size of 5x5, in particular to an array-size of 4x4 or 6x6. By the proposed method 
5 advantageously the transfers of the red, green and blue reconstruction-filters are matched in 
such a way that less or no greenish interferences will occur at higher diagonal frequencies as 
it is the case with conventional filters. The filters are free of post-stamp-border looking 
artifacts and free of color- and luminance-edges. Further all generated luminance signals are 
free of aliasing and distortion by means of the so called luminance-white-compensation in 

10 combination with the filter weights. 

The proposed method offers the possibility to apply several aliasing free 
contour-filters which may be chosen as a function of the optical transfer of the camera. In 
particular an array size to which the contour-reconstruction-filter is applied to exceeds the 
size of an array a color-construction-filter is applied to. Advantageously the contour- 

15 reconstruction-filter is applied to an array size of 4x4 or 6x6. 

In a further preferred configuration the reconstruction-filter is a luminance- 
reconstruction-filter and the pixels of the array are added together in one white pixel being 
the output-pixel. Most preferred the green-parameter or a number of green parameters are 
chosen in dependence of a sensor matrix of the image sensor. Moreover the green parameter 

20 or a number thereof may be chosen in dependence of an optical transfer of an optical system 
providing an image signal to the image sensor. Thereby the RGB-color-signals are 
advantageously reconstructed with filter weights that can be chosen as a function of an 
optical transfer of a camera in combination with the heaviness of the sensor matrix. Most 
preferred two green parameters are provided. 

25 The basic concept of luminance reconstruction as described above implying 

with one or more of the preferred configurations is in the following referred to as the "white 
compensated luminance-reconstruction" for RGB-Bayer-image-sensors, or simply "RGB- 
reconstruction". The green-parameters are also referred to as "smartgreen-parameters". Using 
the smartgreen-parameters also with a red-and/or blue-pixel as defined by the proposed 

30 method and the further developed configurations thereof will be referred to as the 

"smartgreen3 "-reconstruction method. Particular ways of determining the green-parameters 
are also described in WO 99/04555 and EP 01200422,2 and may be applied and used within 
smartgreen3 as well. 
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The specific kind of arranging filters and filtersize of the proposed method 
results in an aliasing tree signal and in particular also free from green-green differences. 
Details will be further outlined in chapters 2 and 3 of the detailed description with regard to 
preferred embodiment and with reference to the (hawing. 
5 The proposed white compensated concept advantageously results in an 

aliasing free luminance signal, even at the multiples of the sample frequency in case of a 
camera without optical low pass filter. Moreover this white compensated luminance signal is 
free of signal distortion. The basic method and apparatus as proposed are adapted to offer a 
broad and flexible extension. It is possible to offer a variety of several reconstruction-filters 
10 which may be chosen and adjusted e.g. in dependence of an optical transfer of an optical 
system and/or a sensor matrix of an image sensor. The proposed method and apparatus are 
capable to maintain a rather independency of optical low pass filters. This is particular 
advantageous as camera designs of potential customers may vary. The proposed method and 
apparatus are capable to implement adjustable false-color-filters in various ways and in a 
1 5 simple manner as will be outlined below. 

Continued developed configurations of the method are described in the 
dependent method claims. The proposed apparatus may be improved with respective means 
for executing the method. 

In particular it is preferred that a center-oulput-pixel of a second filter 
20 subsequent to a first filter is positioned in phase with the output-pixel, in particular the 
center-output-pixel is centered at the same center position of the array as the output-pixel. 
Most will be further described below. 

Specifically a luminance-reconstruction-filter is applied to pixels of an array 
having an array size of 2x2 or 4x4 or 6x6 or larger if preferred. In a particular preferred 
25 configuration the lummance-recoristruction-filter is applied to an array size of 2x2 or 4x4. 
The filter size may be chosen as a function of the optical transfer. Also the weights for the 
respective filter may be chosen differently. 

Additionally a low pass luminance signal may advantageously be generated by 
low-pass-filters applied to an array size of respectively 4x4 or 6x6. In an advantageous 
30 configuration the 4x4 or 6x6 low pass filter is combined with the 2x2 or 4x4 luminance- 
reconstruction filter respectively to establish one single filter. None of the resulting signals 
suffers from green non-uniformity artifacts caused by the sensor. Further details of the 
luminance-reconstruction-filter may be taken from the detailed description and from a patent 
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application with internal file number ED606638-I, which has been filed on the same day as 
this application and which is incorporated by reference herein. 

Nevertheless, depending on the optical transfer and the matrix, the 
reconstructed RGB signals may still suffer from a rest amount of colored aliasing according 
5 to the Nyquist-theorem. In order to still reduce the rest amount of aliasing, additionally to the 
luminance-reconstruction-filter the color-reconstruction-filter may be applied. In particular 
therefore the color-reconstruction-filter comprises a false-color-filter to eliminate false colors 
from the input 

As proposed by the invention and as outlined above already the contour- 

1 0 reconstruction-filter is applied in parallel with the color-reconstruction-filter. 

Preferably the false-color-filter is applied to a smallest possible array of green- 
pixels. Specifically the false-color-filter comprises steps in which the red- and blue-pixels in 
the smallest possible array of green-pixels are weighted by green-parameters and the 
weighted red and blue pixels are summarized by a median filter together with an average of 

1 5 one or more of the green-pixels in the array. An average of one or more green-pixels in the 
array may be taken by applying an average filter. In particular the average of green-pixels in 
a vertical direction and an average of green-pixels in a horizontal direction may be taken. The 
median-filtered pixels are compared with low-frequency-filtered pixels of the smallest 
possible array of green-pixels, whereby false colors are eliminated from the input 

20 Such scheme advantageously allows for an adjustable false-color-filter, in 

particular the color-reconstruction-filter and/or the false-color-filter is applied to a smallest 
array of green-pixels having preferably an array-size of 3x3. Still, if necessary, such 
predetermined small array may also be of 5x5 size or larger. Such adjustment is independent 
of an optical low-pass-filter. Such array however should comprise at least four pixels, i.e. two 

25 green-pixels, one red-pixel and one blue-pixel. 

Preferably the applied color-reconstruction-filter has an array-size of 3 x 3 or 
5x5. In particular, an array-size of 5x5 is preferred in case of a heavy sensor matrix. 

Further, the green, and also the red and blue, color-reconstruction, i.e. the 
RGB-reconstruction, is fully independent of the 3x3 green false-color-array. The RGB- 

30 reconstruction-filter may be of 3 x 3 or 5x5 size. It is a particular advantage of the proposed 
method that the false-color functions are independent of the color-reconstruction-functions. 

Specifically the or a number of green-parameters may be used in the color- 
reconstruction-filter solely within the false-color-filter. 
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Further a color-reconstraction-filter may be applied to an array size of 3 x 3 or 



in particular to an array-size of 5x5 in case of a heavy sensor matrix. The coefficients of a 
corresponding filter function may be chosen as a function of the optical transfer and matrix. 
In particular this is outlined in a chapter 4 of the detailed description of a patent application 
5 with internal file number ID60663 8-1 which has been filed on the same day as this 

application and which is incorporated by reference herein. Such methods maximize the 
resolution for near-white colors which for average scenes seem to be the most important eye 
catchers. In particular sufficient resolution is provided while the amount for signal distortion 
is minimized. 

10 The rules for defining the filter coefficients of a 3 x 3 and a 5x5 RGB color- 

reconstruction-filter are outlined for a preferred embodiment in particular in a patent 
application with internal file number ID505538-II which has been filed on the same day as 
this application and which is incorporated by reference herein. 



15 in its output a phase to the output of a reconstruction-filter which has been applied 

previously. Advantageously the post-filter is applied subsequent to a false-color-filter, i.e. the 
post-filter is applied to the output of the false-color-filter after the RGB-reconstruction-filter. 
The reconstruction-filter may also be the luminance filter. Specifically the phase is fixed by 
advantageously applying subsequent to a false-color-filter a post-filter of 2x2-array-size by 

20 which a center-output-pixel of a smallest possible array of green-pixels is positioned in phase 
with a white-pixel. The white-pixel itself is centered with respect to the same array as to 
which a luminance-reconstruction-filter has been applied to. In particular signals are RGB- 
reconstructed on basis of a 3 x 3 or 5x5 array size and will get the same phase as the white 
compensated luminance signal due to the 2x2 post filter. It is in particular advantage of this 

25 configuration, that the false color-filter does not influence the phase. The post-filter 

advantageously also eliminates a green-non-uniformity of an image sensor. A preferred 
embodiment of the false-color-filter and particular ways of post-filtering may be taken from 
the above mentioned patent application with internal file number BD606638-II. 



30 5x5 RGB color-reconstruction-filter in a further preferred embodiment, which is referred to 
as a smartgreen4-reconstruction-method, outlined in chapter 5 of the detailed description. 
This allows a simple and efficient processing and is therefore advantageous with regard to 
low-cost applications. 



In a further preferred configuration a post-filter is applied in order to maintain 



Still also, as an alternative, a post filter may be left out for the use of a 3 x 3 or 
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Further advantageously a low pass luminance signal is realized having a 
transfer characteristic as equal as possible to the total RGB transfer. The matching of a 4x4 or 
6x6 low-pass-luminanee-filter with respectively a 3 x 3 or 5x5 green reconstruction filter is 
preferably achieved by a 2x2 post-filter. 
5 In a further preferred configuration a high frequency luminance-filter is 

applied in dependence of an optical transfer of an optical system providing an image signal to 
the image sensor. Thereby a high-frequency aliasing free luminance component can be added 
to the low frequency reconstructed color signals in order to mask color artifacts. A 
corresponding preferred embodiment is described in the detailed description with reference to 
1 0 Figure 2 of the drawing. 

The proposed method is adapted such that the optical transfer of the camera 
does not limit the application of a suitable filter. In other words, arbitrary optical low pass 
filters may advantageously be applied, merely depending on the application of a customer. It 
is even acceptable to apply no optical low pass filter at all. Especially this is achieved by the 
1 5 appliance of a false-color-filter as proposed, which is in particular adjustable independent of 
the optical transfer of the camera or of the sensor matrix. In particular, the false color filter 
can be adjusted as a function of the optical transfer of the camera at one half of the pixel 
frequency. 

Furthermore, various luminance-reconstruction-filters are offered for 
20 appliance. In particular a lximinance-reconstruction-filter is applied to an array size of 2x2 in 
case of no or slight optical low pass filtering and further a respective luminance- 
reconstruction-filter is applied to an increased array size of 4x4 or 6x6 upon heavier low pass 
filtering. 

Furthermore, various color-reconstruction-filters are offered for appliance. In 
25 particular a 3x3 color-reconstruction-filter is applied in case of a 2x2 luminance- 
reconstruction-filter or a 5x5 color-reconstruction-filter is applied in case of a 4x4- 
liuninance-reconstruction-filter. 

Furthermore, various contour-reconstruction-filters are offered for appliance. 
In particular a 4x4-contour-reconstruction-filter is applied in case of a 3x3-color- 
30 reconstruction-filter and further a 6x6-contour-reconstruction-filter is applied in case of a 
5x5-color-reconstruction-filter. Advantageously the proposed method allows appliance of 
reconstruction-filters, which are adjustable as a function of the optical transfer of an optical 
system, e. g. of a camera or a sensor matrix. Therefore, depending on the chosen color- 
reconstruction-filter the output does hardly or not suffer from artifacts. A further advantage 
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is, that the green-nonuniformity caused by a sensor is diminished. It is mostly a 5x5 green 
reconstruction filter that suffers from green-green differences. In particular by the aid of a 3 x 
3-green-reconstruction-filter in combination with the 2x2-post-filter such green- 
nonuniformity is removed or diminished. 
5 A preferred derivate of the proposed method is of particular interest and 

dedicated for low cost applications. It has been already mentioned above as smartgreen4 and 
combines the appliance of a 3x3-color-reconstruction-filter and a 5x5-contour-reconstmction- 
filter. In particular subsequently a color-reconstructed and a contour-reconstructed signal are 
added for further processing. Smartgreen4 advantageously offers a low power consumption 
10 and a minimum of hardware. Further details are outlined in chapter 5 of the detailed 
description. 

As a particular preferred and advantageous configuration, the white 
compensated luminance reconstruction scheme of smartgreen as proposed above may be 
combined with a unique 5x5 aliasing and distortion free contour-reconstruction-filter. The 
15 contour-reconstruction-filter may be executed parallel to the color-reconstruction-filter. Such 
combination offers a strong reduction of artifacts in comparison with prior art filter schemes. 
Most advantageously such reconstruction scheme is based on the use of a 6x6 filter array. It 
may also incorporate several aliasing and distortion free parallel contour signals based on that 
array with the same performance as the above mentioned unique 5x5 aliasing free contour- 
20 reconstruction-filter. 

In the whole processing chain as described above, i.e. the - RGB- 
reconstruction, in particular comprising 

- the implementation of a post-filter for phase matching; 

- the luminance-signal processing; 

25 - the color-reconstruction-signal processing, in particular comprising the implementation of 
a false-color-filter and in particular a further post-filter; and in parallel thereto 

- the contour-signal processing, 

the amount of signal distortion is limited to an extremely low level. 

This also holds for a final JPEG-conversion. In a preferred configuration as 
30 well column- and also row-wise processing may be performed to execute the smartgreen3- 
reconstruction-algorithm according to the proposed method. Such processing advantageously 
reduces the amount of internal memory and the amount of data swapping to and from an 
external memory. This will support processing effectivity and speed. Such measure also 
works if all data transfer is rotated by 90°. 
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The proposed method is advantageously executed on an apparatus as proposed 
above, in particular on a computing system and/or a semiconductor device. Such system may 
advantageously comprise an intermediate memory interface located between an image sensor 
and a processing chip. Thereby advantageously the length and the number of rows of a pixel 
5 array to be filtered is no longer restricted, also the amount of data swap to and from an 
external memory of course should not delay the processing time too much. Consequently, 
still real-time processing is possible. The computing system may be any kind of processor 
unit or system or computer. 

Also real-time processing may preferably be performed without any memory 
1 0 using intermediate interface. 

However, in this case, for cost reasons, the total amount of available row 
delays may be limited, in particular to two. This may result in that only three vertical taps are 
available for the RGB reconstruction as well as for the realization for the contour signal. 

Further the invention leads to a computer program product storable on a 
1 5 medium readable by a computing system comprising a software code section which induces 
the computing system to execute the method as proposed when the product is executed on a 
computing system, in particular when executed on the computing system of a camera. 

The invention will now be described with reference to the accompanying 
20 drawing. The detailed description will illustrate and described what is considered as a 
preferred embodiment of the invention. It should of course be understood that various 
modifications and changes in form or detail could readily be made without departing from the 
spirit of the invention. It is therefore intended that the invention may not be limited to the 
exact form and detail shown and described herein nor to anything less than the whole of the 
25 invention as disclosed herein and as claimed hereinafter. Further the features described in the 
description and the drawings and the claims disclosing the invention, may be essential for the 
invention considered alone or in combination. 

The detailed description accompanying the drawing provides the following 



30 



chapters: 

1. Processing flow of white compensated luminance reconstruction method 

2. The white compensated luminance signal 

3. RGB color reconstruction 

4. Parallel aliasing free contour filtering 
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5. Smartgreen4. 

6. Conclusions 

The Figures of the drawing show preferred embodiments of the invention and 
are enumerated as follows: 
5 Figure 1 : Location of RGB- and contour-reconstruction in a memory based 

architecture; 

Figure 2: Basic block diagram of the smartgreen3 reconstruction (particular 
information to the block regarding the multiplication of the R- and B-colors with the 
smartgreen parameters may be taken from WO 99/04555); 
10 Figure 3: Realization of the white compensated luminance pixel Yn; 

Figure 4: Several white compensated luminance reconstruction filters; 

(Figures 3 and 4 illustrate the white-compensated-luminance signal of a 
preferred embodiment which is advantageously free of green-green differences. Green-green 
differences may be removed by a restoration of a green uniformity of a Bayer image which 
15 allows to eliminate green-green differences in the green signal of the sensor with 

maintenance of a Laplacian (i.e. smartgreen) RGB-reconstruction method and without visible 
resolution losses. Also they may be removed by preventing green non-uniformity in the 
parallel contour signal of RGB Bayer image sensors, which may be performed by developing 
a two-dimensional parallel contour filter that eliminates the green-green differences caused 
20 by the image sensor.) 

Figure 5: Three 4x4 contour filters; 

Figure 6: Three 6x6 contour filters; 

Figure 7: Further simplification of the smartgreen3 block diagram of Figure 2 
by using one contour filter; 
25 Figure 8: Transfer characteristics of the three 4x4 contour filters of Figure 5; 

Figure 9: Transfer characteristics of the three 6x6 contour filters of Figure 6; 

(Figures 5 to 9 illustrate a preferred embodiment capable to provide aliasing 
free parallel contour signals. This may be achieved by either parallel color- and contour 
processing for Bayer image sensors, which allows to use the green signal from the image 
30 sensor for generating a two-dimensional contour signal in parallel with the RGB color 
reconstruction. The advantage is that no extra row delays are needed as it is the case with 
serial contour processing. An aliasing free contour processing for a RGB Bayer image sensor, 
which is based on a unique 5x5 parallel contour filter that, has no need for an optical low 
pass filter and has a zero throughput at the first RGB sample frequency. Its signal distortion 
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is almost zero, resulting in a contour signal without visible artifacts. The so far known 
contour filters amplify the back folded and undesired frequencies outside the Nyquist domain 
of the sensor as well. This will lead to distortions and as a consequence a better visibility of 
the unwanted aliasing components in the picture. This unique 5x5 filter prevents those 
5 aliasing artifacts and moreover eliminates the green-green differences caused in the green 
channel of the image sensor.) 



Figure 13: Details of smartgreen4 showing 3x3 RGB reconstruction in 
combination with 5x5 aliasing free contour reconstruction; 

Figure 14: Top-left: 3x3 RGB reconstruction with a 5x5 aliasing free contour 
filter, top-right: an aliasing free contour-filter only, bottom left: a 3x3 RGB reconstruction 
15 with a 5x3 contour-filter, bottom-right: a 5x5 contour-filter only; 

Figures 12 to 14 illustrate a preferred embodiment i.e. a smartgreen4-method 
with 3x3 and 5x5 filter weights of smartgreen3 as shown in Figures 10 and 1 1, wherein there 
is no need for a 2x2 post filter. 



20 1. Processing flow of the white compensated luminance reconstruction method 

In Figure 1 , a part of a general architecture of an integrated circuit with a 
central bus and an external memory interface is shown. A sensor signal is offered via the 
central bus to the external memory. For the realization of the RGB-color-reconstruction and 
of the parallel contour-reconstruction the sensor data is retrieved from the external memory 

25 via the central bus to the reconstruction. After reconstruction the data are sent directly to the 
processing block or sent back to the external memory. 

The processing block contains more or less standardized camera functions as 
the matrix, white balance, knee and gamma. Sending the reconstructed data directly to the 
processing is an important issue because in order to obtain a fast execution time for still 

30 pictures or for video data, the amount of time consuming data swap from and to the memory 
should be limited. 

In Figure 2 a more detailed block diagram of a preferred embodiment of the 
proposed method, hereinafter referred to as "smartgreen3 "-signal-reconstruction, is shown. 
Via the central bus the sensor data is sent from the external memory in small packets to a 



10 



Figure 10: 3x3 green reconstruction for center green is present; 
Figure 11: 3x3 green reconstruction transfer for center green is absent; 
Figures 10 and 11 illustrate a preferred embodiment; 
Figure 12: Basic block diagram of the smartgreen4-reconstruction; 
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small internal memory array in the reconstruction block. From this [1Sx64Hx6V] array in 
Figure 2, containing 1 Signal (16 bits) and 64 Horizontal by 6 Vertical pixels (768 bytes) as 
an example, the raw sensor data can be randomly retrieved for reconstruction. In particular, a 
row of sensor data can be processed in multiples of (64-2*ho) pixels, where ho is the 
5 horizontal offset of the filter array. For an nxm reconstruction array, where "n" are horizontal 
and "m" are vertical pixels, the offset is ho=n div 2 f, div M means rounding in the direction of 
zero to the nearest integer. Therefore, it holds ho=l for n=3, ho=2 for n=4 or n=5 and ho=3 
for n-6. The first pixel that can be reconstructed is at position 1+ho, the last one at position 
N-ho, where N is the total number of pixels in a sensor row. Reconstructing a complete 

10 sensor row requires N/(64-2*ho) packages to be sent to the reconstruction block. In the lower 
part of Figure 2, the RGB color signals are reconstructed with the raw sensor data. The (low) 
frequency transfer, by means of the choice of the filter weights, depends on the optical 
transfer of the camera. Whether the false color killer and the 2x2 post-filters are used or not 
also depends on the optical transfer. In the upper part of Figure 2 the R and B pixels are 

1 5 multiplied with the smartgreen parameters. The smartgreen parameters may e.g. be retrieved 
according to the method disclosed in WO 99/04555 or in EP 01 200 422.2. With this specific 
signal three aliasing free luminance signals are reconstructed: the contour signal, the white 
compensated luminance signal Yn and a low frequency luminance signal Ylf. The latter has 
about the same transfer characteristic as the reconstructed RGB signals. By subtracting the 

20 low frequency signal Ylf from the white compensated luminance signal Yn, a high frequency 
luminance component is generated: (Yn-Ylf). It is important to be aware of the fact that in 
the preferred embodiment, in order to prevent undesired false colors at higher frequencies, 
the (Yn-Ylf) signal if ever possible should not earlier be added to each color signal then after 
the matrix and white balance functions in the processing block. The very same counts for the 

25 contour signal. Preventing the mentioned undesired false colors is the reason why the total 
output of the reconstruction block consists of four or respectively three signals. Besides the 
fastest possible execution time this is a second reason why it is preferred to send all signals 
directly to the processing block. A third reason can be that it makes twice a [4Sx64Hxl V] 
internal memory superfluous, one in the reconstruction block for sending the four signals to 

30 the external memory and one in the processing block for retrieving them again. The 

[4Sx64HxlV] internal memory stands for the storage of four (16 bit) signals of (64-2*ho) 
horizontal pixels of one vertical row, containing a total of 640 bytes when rounding to 64 
pixels. 
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When maximum flexibility of the design is required, e.g. for a time consuming 
reconstruction and/or processing by means of specific software via the CPU, then the two 
[4Sx64HxlV] internal memories, one for the reconstruction and one for the processing, 
should be applied. 

5 At the top of Figure 2 the aliasing free contour is realized followed by the 

overshoot control processor which prevents over- and undershoots at lower frequencies. 
Two-dimensional sharpness improvement using a two-dimensional step transient signal is 
achieved by the realization of a two-dimensional detection signal which is suited for 
controlling overshoots. This so called step transient signal can be used for several overshoot 

10 (and undershoot) control methods, and results in a very attractive sharpness improvement 
without exaggerated and unnatural looking overshoots. It is allowed to add the contour and 
the (Yn-Ylf) signal to a single signal that is sent to the internal [4Sx64Hxl V] memory. 
The following chapters may be summarized as follows. 
In chapter 2, the realization of an aliasing free and zero distortion luminance 

1 5 signal Yn with a 2x2 and 4x4 filter array is described. Details of a luminance-reconstruction- 
filter are disclosed in particular in chapter 2 of the detailed description of a patent application 
with internal file number ID60663 8-1 which has been filed on the same day as this 
application and which is incorporated by reference herein. Also low frequency luminance 
signals with a 4x4 and a 6x6 low pass filter array are explained in chapter 3 therein. The 

20 realization of the low frequency RGB reconstruction with or without a false color killer, is 

further described. Details of a color-reconstruction-filter are disclosed in particular in chapter 
3 of the detailed description in a patent application with internal file number DD606638-II 
which has been filed on the same day as this application and which is incorporated by 
reference herein. The realization of a 5x5 aliasing free contour-filter is described in chapter 2 

25 and 4x4 and 6x6 aliasing free contour signals are described in chapter 4 in the following. 

The concept proposed in this application and the filters described herein may 
also be adapted in a flexible way as a function of the optical transfer and sensor matrix. This 
is also described in detail in chapter 4 of the detailed description of above patent application 
with internal file number ID606638-I. 



30 



2. The white compensated luminance signal 

The white compensated luminance signal is based on the calculation of the 
smartgreen parameters. Examples are shown in WO 99/04555 and EP 01 200 422.2. The 
SmartGcntrlR and SmartGcntrlB parameters are referred to as: wbr and wbb respectively. 
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Figure 3 shows that the red and blue pixels are multiplied with the smartgreen parameters. 
Next the four pixels are added together resulting in a white compensated luminance pixel Yn. 
The center of pixel Yn is shifted half a pixel tothe right and to the bottom, considering the R 
pixel as the first one. The consequence is that all other signals that have to be reconstructed, 
5 i.e. red-green-blue and contour, should get the same center position as this Yn signal. 



same as for the 5x5 aliasing free contour signal. An aliasing free contour- signal for RGB 
Bayer image sensors is based on a unique 5x5 parallel contour filter that, without the need for 
an optical low pass filter, has a zero throughput at the first RGB sample frequency. Its signal 

10 distortion is almost zero, resulting in a contour signal without visible artifacts. The so far 

known contour filters amplify the back folded and undesired frequencies outside the Nyquist 
domain of the sensor as well. This will lead to distortions and as a consequence a better 
visibility of the unwanted aliasing components in the picture. This unique 5x5 contour filter 
prevents those aliasing artifacts and moreover eliminates the green-green differences caused 

15 in the green channel of the image sensor. Particular advantages are: 

1 . Without the need of an OLPF (optical low pass filter), the Yn signal has a zero 
throughput at the first RGB sample frequencies, so it will cause no aliasing at those points. At 
the second and higher multiples of the sample frequencies the throughput is low, but the low 
pass of the lens and the modulation transfer function (MTF) of the sensor will be effective 

20 there as well. 

2. Hie signal distortion is almost zero, resulting in a luminance signal Yn without 
visible artifacts. 

In the following it will be explained why a 2x2 white compensated luminance 
filter is particularly advantageous. 

25 1 . The 2x2 unity is needed for the elimination of the modulated color information 

in the sequential RGB Bayer color signal of the image sensor and the elimination of the green 
non-uniformity caused by the image sensor. Preventing green non-uniformity in the parallel 
contour signal of RGB Bayer image sensors and is achieved by a method for developing a 
two-dimensional parallel contour filter that eliminates the green-green differences caused by 

30 the image sensor. Parallel contour processing for Bayer image sensors allows to use the green 
signal from the image sensor for generating a two-dimensional contour signal in parallel with 
the RGB color reconstruction. 



serial contour processing. It has to be noted that whatever OLPF-type has been used, it does 



The advantages of the white compensated luminance signal Yn are the very 



The advantage is that no extra row delays are needed as it is the case with 
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neither influence this color modulation in large saturated color areas nor the green non- 
uniformity. 

2. The multiplication by the smartgreen parameters is needed in order to undo the 

effect of a matrix, wherein the sum of the coefficients of the matrix is not equal to unity, 
5 and/or the effect of the environmental color temperature. 

These are the two reasons why the smartgreen parameters are calculated. If 
those parameters are not unity, the color sensor will not act as a black and white sensor. 

A slight disadvantage of the 2x2 white compensated luminance signal Yn may 
be that the resolution will be reduced when an optical low pass filter (OLPF) is applied. 

1 0 However this can be compensated by applying 4x4 filters for the reconstruction of Yn. 
Depending on the amount of the low pass transfer of the OLPF, a Yn filter with heavier 
weights can be chosen. On the left hand side of Figure 6 the already described 2x2 Yn filter 
is shown. In the middle and on the right hand side two 4x4 Yn filters are shown. The right 
one is the 'heaviest' filter. Besides for compensating the transfer loss of an OLPF they also 

1 5 can be applied for compensation of the transfer loss of the lens. 

The 4x4 Yn filters restore the green uniformity and eliminate the modulation 
in colored areas as well. The green uniformity restoration can be achieved by preventing 
green non-uniformity in the parallel contour signal of RGB Bayer image sensors which may 
be performed by developing a two-dimensional parallel contour filter that eliminates the 

20 green-green differences caused by the image sensor. Parallel contour processing for Bayer 
image sensors allows to use the green signal from the image sensor for generating a two- 
dimensional contour signal in parallel with the RGB color reconstruction. The advantage is 
that no extra row delays are needed as is the case with serial contour processing. As a rule 
one may say: The subtraction of neighbor diagonal filter coefficients should result in a zero 

25 contribution. This will average and, as a consequence, eliminate the green-green differences 
of the neighbor green pixels. 

The combinations of the negative coefficients always result in an addition of 
two green and a red a blue pixel, and also have the advantage of a zero transfer for the color 
modulation. It is to be noted that depending on the combination of the total optical transfer 

30 and the heaviness of an arbitrary 4x4 Yn filter, finally undesired over- and undershoots can 
occur. The choice of the coefficients can only be done when the optical transfer is known, for 
example by shooting and processing a zone plate scene. 



3. RGB color reconstruction. 
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The reconstruction method described in this preferred embodiment should be 
suited for cameras with an arbitrary optical low pass filter (OLPF) or with no OLPF at all. 
The choice of an appropriate OLPF is not a simple matter. In most cases it is a compromise 
between the loss of sharpness and the acceptance of a certain amount of aliasing or preferably 
5 no aliasing at all. Relevant parameters are: 

1. the optical transfer of the lens; 

2. the modulation transfer function (MTF) of the sensor, including a possible 
pixel microlens, determining light sensitive parameters; 

3. the sensor matrix, determined by the color array of the sensor. An extra 
1 0 amount of aliasing may be caused by the heaviness of the matrix and influence the 

specification of the OLPF transfer. 

4. a given reconstruction algorithm, which of course results in a certain transfer 
characteristic as well, causing the need for more or less aliasing reduction by means of the 
OLPF. 

15 If one of those four parameters in a camera has been modified, one should 

consider the adaptation of an OLPF. Here a problem arises which is that usually an OLPF is 
specified without taking all four mentioned parameters into account Therefore, in the 
preferred embodiment it is tried to anticipate on an existing or a preferred camera design, i.e. 
the optics (pt 1) and the sensor (pt. 2 and 3), a reconstruction method that offers some 

20 flexibility (pf. 4). The transfer of the reconstruction filters can be changed by choosing 
certain filter weights, or the possibility to apply a false color killer. 

In chapter 5 a particular preferred RGB color reconstruction filter is explained: 
using a 3x3 array. Also a 5x5 array may be used for a RGB color reconstruction filter as 
described in a chapter 3.6 of the detailed description of a patent application with internal file 

25 number ID606638-II, which has been filed on the same day as this application and which is 
incorporated by reference herein. Both can be applied with (or without) a false color killer. 

4. Parallel aliasing free contour filtering 

Similar to the unique 5x5 aliasing free contour filter mentioned above, the 4x4 
30 and 6x6 aliasing free contour filters have a zero throughput at the first RGB sample 

frequency independent of the optical low pass transfer. Also the signal distortion is almost 
zero, as for the (Yn-Ylf) signal. This means that the aliasing free contour signal has no visible 
artifacts, because there is no back folding of undesired frequencies outside the Nyquist 
domain of the image sensor. Until now only one unique 5x5 aliasing free contour filter exists, 
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however there are many 4x4 and 6x6 aliasing free contour filters possible, which all 
eliminate the green non-uniformity as well. In Figure 5 three 4x4 and in Figure 6 three 6x6 
contour filters are shown. From the left to the right the 4x4 contour filters have an increasing 
throughput for high frequencies. In Figure 8 from the top to the bottom the three transfer 
5 characteristics of the 4x4 filters of Figure 4 respectively are shown. In Figure 9 the three 
different transfer characteristics of the 6x6 filters of Figure 6 respectively are shown starting 
at the top with the left 6x6 filter of Figure 6. The middle contour filter of Figure 6 has less 
amplification at the higher frequencies as compared to the left and the right one. This can 
clearly be seen in Figure 9. 

10 By leaving out the upper and bottom row of the 6x6 filters in Figure 6, 6x4 

filters are obtained with a sigmaContour of respectively 68, 52 and 84. The reason this is 
mentioned is because it offers the possibility of a smartgreen3-method with 4 rows or 
respectively 6 rows. On purpose the first 4x4 filter of Figure 5 has the very same weights as a 
first (Yn-Ylf) filter, just as the first 6x6 filter of Figure 46 is the same as a further (Yn-Ylf) 

1 5 filter. Again there is a possibility to combine the filter weights of the (Y n- Ylf) and the 

contour filter, resulting in a further simplification of the smartgreen3-method design. If done 
so, then the coring or a local noise reduction (LNR) or an overshoot control processing 
(OCP) may be performed on the total obtained contour signal, or on a part of it, as it is shown 
in Figure 7. The dashed part illustrates that a part of the contour signal, without LNR and 

20 OCP, can be applied as a (Yn-Ylf) signal. From a performance point of view there are no 
objections to do so. It is still possible to switch off the contour signal for demonstration 
purposes and, if switched on, to control its parameters. This possibility has same advantages 
with regard to a variety of applications. The only disadvantage of the block diagram of Figure 
7 is that trying out another contour filter transfer, has consequences for the (Yn-Ylf) signal as 

25 well. Therefore, if a convenient adjustment and having some flexibility, is given priority it 
may be preferred in an alternative not to combine the (Yn-Ylf) filter and the contour filter. 

As already stated, the choice of the (Yn-Ylf) filter is not too critical. 
Therefore, it does not make sense to describe how to combine its filter weights with the 
weights of the contour filter in order to obtain a single filter. 

30 The choice of the contour filter type and the adjustment of the amount of 

contour depends on the personal taste at judging several pictures. The only directive that can 
be given is that it is rather logical to apply a 4x4 contour filter when a 3x3 color 
reconstruction has been chosen and a 6x6 contour filter in case of a 5x5 color reconstruction. 



PHNL0205 83EPP ^ 

20 25.06.2002 
Further, while adding contour, special attention should be paid to the desaturation of colored 
edges. 

In the following, two very sophisticated contour processing features as 
overshoot control processing (OCP) and local noise reduction (LNR) is explained. A general 
procedure for OCP may use a continuous luminance signal as it is available for example in 
TV-applications. In such a procedure, two dimensional sharpness improvement using a two- 
dimensional step transient signal is achieved by the realization of a two-dimensional 
detection signal that is suited for controlling overshoots. This so called step transient signal 
can be used for several overshoot, and also undershoot, control methods, resulting in a very 
attractive sharpness improvement without exaggerated and unnatural looking overshoots. In 
multiplexed RG/RB sensors special attention should be paid to the most important goal of the 
smartgreen3 processing, being the prevention of visible artifacts. Regarding the LNR, a 
method for a very specific two-dimensional contour filter can be applied. By such a method a 
contour with local noise reduction for digital electronic cameras is achieved by the realization 
of a very specific two-dimensional contour filter that is capable to reduce the amount of noise 
as well as the sharpness by means of a two-dimensional standard deviation function. An 
arbitrary contour filter, as applied with smartgreen3 and described above, preventing visible 
artifacts is also possible. 

In this chapter a smartgreen3 derivate will be described which is referred to as 
smartgreen4. It consists of a 3x3 color reconstruction, does not use the (Yn-Ylf) signal and 
should preferably only be applied with two possible contour filters, a 5x5 or a 5x3 one. It is 
particular suitable for low cost applications. 

5. Smartgreen4 

A very fast processing method with a very acceptable result, better than the 
smartgreenl/2 methods, is offered by the combination of, on the one hand, the 3x3 RGB 
reconstruction as described in a chapter 3.1 of the detailed description and, on the other hand, 
the 5x5 aliasing free contour filter as mentioned above. The mentioned unique 

The mentioned unique 5x5 parallel contour filter has without the need for an 
optical low pass filter, a zero throughput at the first RGB sample frequency. Its signal 
distortion is almost zero, resulting in a contour signal without visible artifacts. The so far 
known contour filters amplify the back folded and undesired frequencies outside the Nyquist 
domain of the sensor as well. This will lead to distortions and as a consequence a better 
visibility of the unwanted aliasing components in the picture. This unique 5x5 contour filter 
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prevents those aliasing artifacts and moreover eliminates the green-green-differences caused 
in the green channel of the image sensor. 

The transfer characteristics of the RGB color-reconstruction-filters are shown 
in Figures 10 and 1 1 and have better matching characteristics for a 3x3 green-reconstruction- 
5 filter as conventional filters. In the upper-left corner the filter weights are displayed. 

Although a 2x2 post-filter will eliminate green-green differences, also this 
green-reconstruction-filter will already do that. A method for developing a two-dimensional 
parallel contour filter that eliminates the green-green differences caused by the image sensor 
can be applied, 

10 As a rule, "the subtraction of neighbor diagonal filter coefficients should result 

in a zero contribution. This will average and as a consequence eliminate the green-green 
differences of the green-pixels 11 . The use of this rule has also been explained in the 
description of the white compensated luminance signal in chapter 2. 

The advantages of better matching green filters are that: 

15 - they will cause less greenish interferences at higher diagonal frequencies 

- less false color killer artifacts will occur, 

- the signal is free of artifacts at edges. 

It is to be noticed that a difference with the smartgreenl/2 reconstruction method is that 
20 smartgreen4 like smartgreen3 does not apply the smartgreen signal as a color signal. The 

smartgreen signal is only used for the false color detector as illustrated in Figure 13. The lack 
of artifacts in lower image parts is mainly caused by green transfer characteristics. The filters 
described here are matching better than the filters in conventional reconstruction methods. 
This is a particular advantageous difference, which offers a surprisingly perfect 
25 reconstruction at colored and black and white edges. This makes the green filter combination 
of the Figures 10 and 1 1 very unique and it will be explained in the following how the 3x3 
reconstruction-filter is applied within the smartgreen4-reconstruction. 

The method is referred to as smartgreen4 and it offers all the mentioned 
advantages of both, reconstruction methods and also methods for the elimination of green- 
30 green differences. Because it is fast it can be applied for video signal reconstruction or, 

depending on the application, for still images as well. It is an interesting option to implement 
the possibility of this unique combination in a smartgreen3 chip design. Also for dedicated 
applications, requiring low power and a minimum of hardware, as for example with CMOS 
sensors, this combination seems to be a very advantageous embodiment For dedicated high 
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volume and low cost CMOS applications again fixed row delays can be applied in order to 
avoid external memory. In Figure 12 the basic block diagram for such an application is 
shown with an internal memory of [lSxrowx4V], i.e. four dedicated row delays. In Figure 13 
a more detailed diagram of the reconstruction is shown. 
5 The false color detector acts with a smartgreenl signal and applies 

respectively for the presence of green and for the absence of green. In case of the absence of 
green the detector should only be applied using a 5x5 array. Therefore here it has been 
chosen for the simplest version, using the smartgreenl signal for the center row only. Four 
row delayed signals are offered to the 5x5 aliasing free contour reconstruction filter. The 
10 actual sensor signal contains the data of row 1 . No local noise reduction, or overshoot control 
has been applied, however this is of course possible. Only a simple coring circuit reduces the 
noise. 

At the left-top of Figure 14 the result of the above described processing is 
shown. For all four quadrants of the artificial zone plate the same optical low pass filter has 
15 been applied as well as the matrix of the Philips FT19 sensor. 

At the top-right the 5x5 aliasing free contour signal of the zone plate is shown 
(with a gain of 0.7). As can be seen it is completely free of interferences. 

At the bottom-right the 3x3 RGB reconstruction by use of filters of Figures 10 
and 1 1 is shown combined with a very specific 5x3 parallel contour filter, which 
20 advantageously eliminates the green-green differences caused by the image sensor. Also for a 
further price reduction such a filter is very interesting because then only two instead of four 
row delays are needed. As can be seen the performance is less good. The sharpness has 
become asymmetrical due to the loss in the vertical direction and there are more 
interferences. Further more greenish aliasing occurs around the RGB sample frequency in the 
25 outer bottom-left comer. At the bottom-right only the 5x3 contour signal is shown (gain 1 .0) 
which can be easily compared with the above lying 5x5 aliasing free contour. Most 
preferably, the contour signals also have been added to the RGB signals after the matrix and 
white balance processing. 

For the processed results on the left hand side the above described false color 
30 killer has been applied. 

Both applications shown in Figure 14, at the top with the 5x5 aliasing free 
contour and at the bottom with the specific 5x3 contour, should be judged as smartgreen4 
processing methods. 
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6. Conclusions 

The following particular advantages are achieved with the smartgreen3 
reconstruction-method compared with smartgreenl and smartgreen2: 

It is a flexible design. Depending on the heaviness of the sensor matrix a 
5 choice can be made between two color reconstruction filters. As a consequence the (low 

pass) luminance filters can be defined application specific. Several high frequency luminance 
filters can be chosen and adjusted as a function of the optical transfer of the camera. 

• If the false color killer is needed, then depending on the chosen color 
reconstruction, it does hardly or not suffer from visible artifacts (black and white dots). The 

10 reason is that a stronger low pass filtered color reconstruction is used than with smartgreenl 
or smartgreen2. 

• There are no artifacts at all due to the smartgreen reconstruction for the simple 
reason that the smartgreen concept is used only for the false color detection and not as a color 
signal. In order to compensate the loss of resolution, a high frequency luminance signal is 

15 generated. 

• Depending on the chosen color reconstruction, the bandwidth of the back 
folding colored aliasing is smaller than with smartgreenl and smartgreen2. 

• The transfers of the red, green and blue reconstruction filters are matched in 
such a way that less or no greenish interferences will occur at higher diagonal frequencies as 

20 it is the case with the smartgreenl and smartgreen2 filters. 

• The color reconstruction filters are free of post stamp border looking artifacts 
at color and luminance edges. 

All generated luminance signals are free of aliasing and distortion by means of 
the so called luminance white compensation in combination with the filter weights. 
25 * Smartgreen3 offers several aliasing free contour filters which can be chosen as 

a function of the optical transfer of the camera. 

All filters of the smartgreen3 design, presented here, eliminate the green non- 
uniformity caused by the sensor. The 5x5 green reconstruction filter finally does it with the 
help of the 2x2 post-filter. 



30 



In summary, due to its flexibility, the lack of signal distortion and its false 
color killer, smartgreen3 is well suited for the reconstruction and processing of digital still 
images and of contiguous video pixels of sensors having a Bayer color filter array. 
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An interesting derivative of smartgreen3 is smartgreen4. By offering a low power 
consumption and a minimum of hardware smartgreen4 is suited for dedicated still and video 
applications. 



PHNL020583EPP ^ 

25 25.06.2002 

CLAIMS: 



1 . A method for signal processing, wherein a sensor signal of an image sensor is 
provided as an input and wherein the input is reconstructed in a filter to establish an output 
for further processing, wherein the filter comprises at least one reconstruction-filter selected 
from the group consisting of: a lurninance-recoiistruction-filter, a red-green-blue-color- 

5 reconstruction-filter and a contour-reconstruction-filter, wherein 

— the input comprises a plurality of pixels, and a pixel provides a color value assigned to at 
least one of the colors red, green or blue, 

characterized by 

— applying the luminance-reconstraction-filter to an array of pixels of predetermined array 
10 • size comprising a number of pixels, wherein at least one of the number of pixels is 

formed by a re-pixel assigned to the color of red, at least one of the number of pixels is 
formed by a blue-pixel assigned to the color of blue, and at least one of the number of 
pixels is formed by a green-pixel assigned to the color of green, 

— weightening the red- and/or the blue-pixel by a green-parameter, and 
15 — summarizing the pixels of the array into one output-pixel, and 

— centering the output-pixel in the array, and 

— applying the contour-reconstruction-filter in parallel with the color-reconstruction filter. 

2. The method as claimed in claim 1, characterized by positioning a center- 
20 output-pixel of a second filter subsequent to a first filter in phase with the output-pixel, in 

particular by centering the center-output-pixel at the same center position of the array as the 
output-pixel. 

3. The method as claimed in any one of the preceding claims, characterized by 
25 applying the luminance-reconstruction-filter to an array-size of 2x2 or 4x4 or 6x6. 

4. The method as claimed in claim 4, characterized by applying a low-pass-filter 
to an array size of 4x4 or 6x6. 
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5. The method as claimed in claim 4 or 5, characterized by combining the 
luminance-reconstruction-filter and the low-pass-filter into one single filter. 

6. The method as claimed in any one of the preceding claims, characterized by 
5 applying the color-reconstruction-filter to an array-size of 3 x 3 or 5x5, in particular has to 

array-size of 5x5 in case of a heavy sensor matrix. 

7. The method as claimed in claim 14, characterized by applying subsequent to a 
false-color-filter a post-filter of 2x2 array-size, to position a center-output-pixel of a 

1 0 predetermined small array of green-pixels in phase with a white-pixel which is centered with 
respect to the same array as that to which the luminance-reconstruction-filter has been 
applied to. 

8. The method as claimed in any one of the preceding claims, characterized by 

1 5 applying the contour-reconstruction-filter, in parallel with the lxtminance-reconstruction-filter 
and by adding their reconstructed signals thereafter. 

9. The method as claimed in any one of the preceding claims, characterized by 
applying the contour-reconstruction-filter to an array-size, which exceeds the size of an array 

20 to which the color-reconstruction-filter is applied to. 

1 o. The method as claimed in any one of the preceding claims, characterized by 

applying the contour-reconstruction-filter to an array-size of 5x5, in particular to an array- 
size of 4x4 or 6x6. 

25 

1 1 . The method as claimed in any one of the preceding claims, characterized by 

offering various luminance-reconstruction-filters for appliance, in particular by applying a 
luminance-reconstruction-filter to an array size of 2x2 in case of no or slight optical low pass 
filtering and/or applying a respective liiminance-recoiistruction-filter is applied to an 
30 increased array-size of 4x4 or 6x6 upon heavier optical low pass filtering. 



12. The method as claimed in any one of the preceding claims, characterized by 

offering various color-reconstruction-filters are offered for appliance, in particular by 
applying a 3x3 -color-reconstruction-filter in case of a 4x4-luminance-reconstruction-filter 
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and/or applying a 5x5-color-reconstruction-filter in case of a 6x6-liuninance-reconstruction- 
filter. 

13. The method as claimed in any one of the preceding claims, characterized by 

5 offering various contour-reconstruction-filters for appliance, in particular by applying a 4x4- 
contour-reconstruction-filter in case of a 3x3-color-reconstruction-filter or applying a 6x6- 
contour-reconstruction-filter in case of a 5x5-color-reconstraction-filter. 

14. The method as claimed in any one of the preceding claims, characterized by 
10 applying a 3x3-color-reconstraction-filter in combination with a 5x5 contour-reconstruction- 
filter, in particular by adding subsequently a color-reconstructed and a contour-reconstructed 
signal for further processing. 

15. An apparatus for signal processing, which is in particular adapted to execute 
15 the method as claimed in claims 1 to 16, comprising an image sensor for providing a sensor 

signal as an input and a filter for reconstructing the input to establish an output for further 
processing, wherein the filter comprises at least one reconstruction-filter selected from the 
group consisting of: a luminance-reconstruction-filter, a red-green-blue-color-reconstruction- 
filter and a contour-reconstruction-filter, wherein 
20 - the input comprises a plurality of pixels, and a pixel provides a color value assigned to at 
least one of the colors red, green or blue, 
characterized in that 

— the reconstruction-filter is adapted to be applied to an array of pixels of predetermined 
array size comprising a number of pixels, wherein at least one of the number of pixels is 

25 formed by a red-pixel assigned to the color of red, at least one of the number of pixels is 

formed by a blue-pixel assigned to the color of blue, at least one of the number of pixels 
is formed by a green-pixel assigned to the color of green and 
the apparatus is further comprising: 

— means for weightening the red- and/or the blue-pixel by a green-parameter, 
30 - means for summarizing the pixels of the array into one output pixel, and 

— means for centering the output pixel in the array and 

— means for parallel processing of the contour-reconstruction-filter and a color- 
reconstruction-filter. 



10 
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16. A computer program product storable on medium readable by a computing 
system, in particular a computing system of a camera, comprising a software code section 
which induces the computing system to execute the method as claimed in any one of claims 1 
to 16 when the product is executed on the computing system, in particular when executed on 
a computing system of a camera. 

17. A computing system and/or semiconductor device, in particular a computing 
system of a camera, for executing and/or storing a computer program product as claimed in 
claim 18 thereon. 

18. A camera comprising an optical system, an image sensor and an apparatus as 
claimed in claim 16 or a computing system as claimed in claim 19. 
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A reconstruction method based on a white-compensated luminance- 
reconstruction and using filter weights referred to as smartgreen-parameters is proposed. An 
aliasing free luminance signal, even at the multiples of the sample frequency, and in case of a 
camera without optical low pass filter is achieved. Moreover this white-compensated- 
luminance-signal is free of signal distortion. The proposed method allows a suitable low pass 
filter to be added or combined and in particular will suited to implement a variety of aliasing 
free color- and contour-filters. The RGB color signals are reconstructed using filter weights 
that can be chosen as a function of the heaviness of the sensor matrix and of the optical 
transfer of the camera. It is proposed to apply a contour-reconstruction-filter in parallel with 
the color-reconstruction-filter. 



Fig. 1 



PHNL020583 



1/14 



16 bit in 



from sensor 



Still or Video 
data input 



CPU 



„ ♦ , 

External 
Memory 
Interface 




RGB and contour 
reconstruction 



Processing 



PHNL020583 



2/14 



" Internal 
1Sx64Hx6V 
memory 



I 

£ 

C 

8 



central 

bus 
Interface 



1 
1 

I 

I 



Internal 
4Sx64Hx1V 
memory 



multiplication 
of RB with 
smart green 
parameters 



sensor data 



parallel contour 




reconstruction 


contour 


overshoot control 





white compensated 


Yn 


luminance 






reconstruction 







low frequency 

luminance 
reconstruction 




gain 
control 



/ 
/ 

J . 


RGB colour 
reconstruction 
+ 

optional 
false colour killer 
& 2x2 post-filters 




— » 

— 






— ► 



reconstructed RGB, {(Yn-Ylf) + contour} data 



to processing block or external memory 



to processing block 



Fig.2. 



PHNL020583 



3/14 



R 


G 


G 


B 



basic RGB Bayer 
array 



wbr.R 


G 


G 


wbb.B 



white compensated 
Bayer array ~~ 



4- 



I 



luminance pixel Yn 
by interpolation 



Fig.3 



PHNL020583 



4/14 































1 


1 










1 


1 















































-1 


-1 








-1 


9 


9 


-1 






-1 


9 


9 


-1 








-1 


-1 



















Fig* % 



PHNL020583 


















-1 


-2 


-2 


-1 






-2 


5 


5 


-2 






-2 


5 


5 


-2 






-1 


-2 


-2 


-1 

















sigmaContour = 20 



5/14 

















-1 


-4 


-4 


-1 






-4 


9 


9 


-4 






-4 


9 


9 


-4 






-1 


-4 


-4 


-1 

















sigmaContour = 36 



Fig.S 


















-1 


-6 


-6 


-1 






-6 


13 


13 


-6 






-6 


13 


13 


-6 






-1 


-6 


-6 


-1 

















sigmaContour = 52 



PHNL020583 ^ 

6/14 







-1 


-1 








-2 


-7 


-7 


-2 




-1 


-7 


18 


18 


-7 


-1 


-1 


-7 


18 


18 


-7 


-1 




-2 


-7 


-7 


-2 








-1 


-1 







sigmaContour = 72 



-1 


-1 


-2 


-2 


-1 


-1 


-1 


-2 


-4 


-4 


-2 


-1 


-2 


-4 


17 


17 


-4 


-2 


-2 


-4 


17 


17 


-4 


-2 


-1 


-2 


-4 


-4 


-2 


-1 


-1 


-1 


-2 


-2 


-1 


-1 


sigmaContour = 6 


8 



Fig. 6 



-1 


-1 


-2 


-2 


-1 


-1 


-1 


-2 


-8 


-8 


-2 


-1 


-2 


-8 


25 


25 


-8 


-2 


-2 


-8 


25 


25 


-8 


-2 


-1 


-2 


-8 


-8 


-2 


-1 


-1 


-1 


-2 


-2 


-1 


-1 



sigmaContour = 100 



PHNL020583 



7/14 



Internal 
1Sx64Hx6V 
memory 




multiplication 
of RB with 
smart green 
parameters 



sensor data 



Internal 
3Sx64HxlV 
memory 



RGB data post-filter 



parallel contour 
reconstruction 
+ 

overshoot control 



contour 



RGB colour 
reconstruction 
+ 

optional 
false colour killer 
& 2x2 post-filters 



(Yn-Ylf) ^ 



Jr A gaIn 
control 



B 



reconstructed RGB, {(Yn-Ylf) + contour} data 



to processing block or external memory 



Fig.* 



PHNL020583 



8/14 




Fie. 8 



PHNL020583 



10/14 




Fig- AO 



PHNL020583 



11/14 




PHNL020583 



12/14 



Internal 
1Sxrowx4V 
memory 



sensor 



signal 



i 

o 



multiplication 
of RB with 
smart green 
parameters 



5x5 parallel 
contour 


contour 


reconstruction 


. — 



3x3 RGB colour 

reconstruction 
+ 

optional 
false colour killer 



G 



B 



reconstructed RGB and contour data 



to processing block 
FigAZ 



PHNL020583 



13/14 



actual signal of rowl 



row 



1 

E 



row3 
row4 
row5_ 



multiplication 
of RB with 
smart green 
parameters 



5x5 

>-] aliasing free 
contour 



conto 




FCsIgmaG 



3x3 green 
reconstruction 
with median filter 




false colour 

detector 
according to 
methodl & 2 



green signals 



row2 



row3 



row4 



3x3 
RGB colour 
reconstruction 



Glf^ 



i 



false colour 

killer 
Rlf=Gtf, Blf=Glf 



Rlf 



Glf 



Blf 



Fig.43 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

^ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 
LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



